SummEval: Re-evaluating Summarization Evaluation

Abstract The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack consensus regarding protocols continue to inhibit progress. We address existing shortcomings methods along five dimensions: 1) we re-evaluate 14 automatic in a consistent fashion using neural model outputs with expert crowd-sourced human annotations; 2) consistently benchmark 23 recent models aforementioned metrics; 3) assemble largest collection summaries generated by trained CNN/DailyMail news dataset share it unified format; 4) implement toolkit that provides an extensible API evaluating across broad range 5) most diverse, terms types, judgments model-generated CNN/Daily Mail annotated both judges crowd-source workers. hope this work will help promote more complete protocol as well advance research developing better correlate judgments.